PHNL010046 



1 16.01.2001 

Green reconstruction for image sensors 



m 



The present invention relates to interpolating missing color values of a pixel in 
an array of pixels. 

In digital cameras a lens is used to obtain light information from a scene. 
These light photons are converted into electrons by an image sensor consisting of a number 
of light sensitive elements. When using digital RGB still cameras, three image sensors are 
used to obtain color sensing for each color red, green and blue. In order to reduce both cost 
and size of the camera, it is possible to use one image sensor having an RGB Bayer filter 
array in which each pixel in the sensor array senses red, green or blue in a predefined pattern. 
Jf 10 This pattern is build up of alternating green, red columns and green, blue columns. When 
M> using one image sensor to sense all three colors, it is necessary to reconstruct the missing 

I pixels before the total image is represented. 

In WO 99/04555 a green reconstruction method for RGB Bayer image sensors 
S has been described. The reconstruction concerns only the green color reconstruction, the red 

15 and blue colors remain reconstructed in a conventional way. The reconstruction of the 
missing green pixel is carried out by means of a median filter that sorts three specific 
variables: two of them are derived from the green color, the third one from the red or blue 
color. A disadvantage of this method is that for high saturated colored edges artefacts which 
look like the border of a postage stamp are introduced. 
20 First a short description of the green reconstruction method of WO 99/04555 

will be given. This method will be referred to as smartgreenl reconstruction. The algorithm 
for smartgreenl is based on the fact that resolution losses are best observed in high frequency 
near white scene parts and less near colored parts. 

With this in mind, the contribution of the red and blue pixel is used to help 
25 determine the reconstruction value of the missing green pixels. The objective of smartgreenl 
reconstruction is to maximize the resolution of the green color. For this purpose a median 
filter algorithm is applied. In Fig. 1 the chosen definition of the green pixels surrounding the 
missing green pixel, i.e. the location which is occupied by a red or blue (R/B) pixel, is shown. 
This is used to declare the following variables: 
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• G12 = (Gl+G2)/2 
G34 = (G3+G4)/2 

Further the variable RBc is defined as the near white adapted red or blue color 
value of the center pixel, defined later, and SmartGcntrlR and SmartGcntrlB are near white 
5 matrix and white balance correction parameters. The calculation of these parameters is 
described below. 

In smartgreenl reconstruction for especially digital signal, processing is 
performed by sorting the data of G12, G34 and RBc in order of magnitude. The center value 
of the sorted sequence, also called the median value, will be applied for the reconstruction of 
10 the missing green pixel. 

smartG = median( G12, G34, RBc) 

For a red and blue center pixel, respectively, the RBc value becomes: 
RBc = SmartGcntrlR * R 
fit RBc = SmartGcntrlB * B 

jj 15 The median filter for green replaces the conventional green reconstruction. 

*2 The conventional red and blue reconstruction method is, however, maintained. 

S*S In Fig. 2 an example of a vertical green-magenta scene color transient is 

shown. In total four different modes are illustrated: A trailing green edge and a leading green 
20 edge and the phase relations to the missing green pixel in the center. 

The green color is illustrated at the top of the Fig.; above the pixel definitions 
and below the definitions the magenta transients are shown. Finally, the corresponding 
projection of green on the imaginary sensor pixels is shown when interpolating using the 
smartgreenl method. 

25 According to the smartgreenl median algorithm, being smartG = median(G12, 

G34, RBc), the result of the reconstructed center green becomes 0.5 for all four modes. 

For the shown vertical colored edges smartgreenl causes a green intensity 
modulation in the vertical direction resembling the border of a postage stamp. It should be 
noted that the same applies to colored edges in the horizontal direction. Fig. 3 shows the 
30 RGB reconstruction result with smartgreenl of a green-magenta scene transient. Green to the 
left and magenta to the right. 



Hi: * 



It is an object of the present invention to provide a method for green color 
reconstruction preventing the above artefacts. To this end, the invention provides a green 
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* reconstruction for image sensors as defined in the independent claims. Advantageous 
embodiments are defined in the dependent claims. 

According to a preferred embodiment of the present invention, the green 
modulation problem can be prevented using a median filter that sorts the four individual 
neighboring green pixels and the center RBc value. The following applies to this 5-pixel 
median filter: smartG = median(Gl, G2, G3, G4, RBc) 

In a preferred embodiment of the present invention, it is determined whether a 
selected pixel forms part of an edge in the image and whether said edge has an angle to the 
vertical line substantially different from 90 degrees. If so, the prior art smartgreenl algorithm 
is used, while the inventive 5-input median is used for substantially vertical edges. The edge 
detection preferably involves the steps of: 

a) forming a first pair of diagonal pixels by selecting a vertical and a horizontal 
neighboring pixel and determining the difference between the color values of these pixels, 

b) forming a second pair of diagonal pixels by selecting a vertical and a 
horizontal neighboring pixel and determining the difference between the color values of these 
pixels, and 

C ) determining the difference between color values of the pixels of the first and 

second pair of pixels. 

Step c) may further comprise a step of determining if the determined 
differences are below the predefined level. This predefined level may be between 3 and 10 
percent of maximum color value, such as between 4 and 8 percent of maximum color, such as 
between 5 and 7 percent of maximum color value. 

These and other aspects of the invention will be apparent from and elucidated 
with reference to the embodiments described hereinafter. 

In the drawings, 

Fig. 1 shows the definition of the green pixels surrounding the missing green 

pixel; 

Fig. 2 shows a vertical green-magenta scene color transient; 
Fig. 3 shows the RGB reconstruction result with smartgreenl of a green- 
magenta scene transient; 

Fig. 4 shows the colored edge reproduction by the 5-pixel median filter; 
Fig. 5 shows the reconstructed RGB transient with the 5-pixel median filter; 
Fig. 6 shows the diagonal direction detection; 
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Fig. 7 shows an RGB Bayer luminance pixel; 

Fig. 8 shows the RGB and contour reconstruction, matrix and white balance of 
an RGB Bayer color camera; and 

Fig. 9 shows a block diagram for parallel RGB reconstruction and contour 

5 filtering. 

With a green-magenta colored scene transient, the green reconstruction result 
of this filter becomes as shown in Fig. 4. The green color is illustrated at the top of the Fig. 
and between the imaginary pixel illustrations the magenta transients are shown above the 
10 corresponding projection of green on the imaginary sensor pixels. The undesired modulation 
at the colored edges has vanished. In Fig. 5 the RGB reconstruction result of the 5-pixel 
median filter is shown. Green is shown on the left side of the Fig. and magenta is shown on 
the right side. 

Unfortunately, the 5-pixel median filter causes some extra distortion in a zone 
|5 15 plate scene, especially in the diagonal direction just outside the green Nyquist domain. It also 
suffers from distorted border artefact if the colored edges are rotated 45 degrees. A direction 
dependent condition can help to reduce or eliminate the diagonal amount of distortion. 

A diagonal direction detection can be performed using the following 

algorithm: 

p. 20 If a first set of two diagonal pixels has approximately the same first green 

W i 

% n value and a second set of diagonal pixel has approximately the same second green value 

being different from said first green value, there is a large chance that it is a diagonal edge, 
see Fig. 6. Written in a software form this becomes: 
if ((abs(Gl-G3)<SGllevel) and (abs(G4-G2)<SGllevel)) xor 
25 ((abs(G3-G2)<SGllevel) and (abs(Gl-G4)<SGllevel)) then select smartgreenl 
else select 5-pixel median filter. 



3** 



m : 

v. K: 



A certain percentage (5-7 percent) of the full-scale amplitude applies to the 
SGlevel. In practice it is about 6.25% (being a value of 16 on a 256 full-scale range). 
30 Referring to the previous declaration of variables as applied for smartgreenl, 

the following applies to the new green reconstruction method according to this invention: 
if ((abs(Gl-G3)<SGllevel) and (abs(G4-G2)<SGllevel)) or 
((abs(G3-G2)<SGllevel) and (abs(Gl-G4)<SGllevel)) then 
smartG = median( G12, G34, RBc ) 
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else smartG = median( GI, G2, G3, G4, RBc) 

To the RBc value applies that: 
RBc = SmartGcntrlR * R 
RBc = SmartGcntrIB * B 

This new method has been tested on colored edges and has been compared 
with the conventional reconstruction. Clearly, the method of the present invention offers the 
best results. 

In the following the above mentioned parameters SmartGcntrlR and 
SmartGcntrIB will be explained. 

For a near white luminance signal in the reconstruction block, derived from 
the RGB pixels of the image sensor, the matrix and white balance parameters have to be 
taken into account. Since the matrix and white balance are located after the reconstruction, 
some adaptation of the incoming red and blue colors is necessary. For that purpose the 
parameters SmartGcntrlR and SmartGcntrIB are used to control the red and blue amplitudes 
in order that they match with green and result in a near white luminance signal Yn. Referring 
to Fig. 7, the following applies to this Yn-signal in case of red and blue pixels: 
Yn[i,j] = SmartGcntrlR * red 

Yn[i+l,j+l] = SmartGcntrlR * blue 

In case of green pixels Yn is equal to green: 
Yn[i+l,j] = green 
Yn[ij+1] = green 

In Fig. 8 a simplified block diagram is shown with the RGB and aliasing free 
contour reconstruction, followed by the matrix and the white balance. This block diagram is 
used to define the parameters in the next formulas for the calculation of SmartGcntrlr/B. 
Light LS from a scene is passed to an RGB Bayer sensor S thru a lens L. An output signal 
from the sensor S is applied to a CDS (correlated double sampling, age (automatic gain 
control) and ADC (analog to digital conversion) processing block 1. An output of the 
processing block 1 is applied to an RGB reconstruction and parallel contour processing block 
3. The processing block 3 outputs reconstructed RGB signals Ri, Gi and Bi, as well as an 
aliasing-free contour signal AFC. The reconstructed RGB signals Ri, Gi and Bi are applied to 
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- a matrix circuit MX that produces signals Ro, Go and Bo, which are applied to a white 
balance circuit WB to furnish output signals Ro', Go' and Bo'. 



A correction of each RGB Bayer color sensor's primary colors into the EBU 
5 primaries, which are accustomed in worldwide television sets and PC monitors, is necessary. 
The correction is realized with a matrix that requires nine multipliers. 
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Ro, Go, Bo are the output RGB signals of the matrix and Ri, Gi, Bi are the input signals. 



10 With the white balance after the matrix, the RGB signals become: 

Ro' — awbR - Ro 
Go=Go 
y Bo = awbB • Bo 

where awbR and awbB are the white balance parameters. (According to the World Gray 
Assumption method (WGA), awbR=totalGreen/totalRed and awbB=totalGreen/totalBlue, 
where totalRed, totalGreen and totalBlue represent the total of the RGB color amplitudes 
jf 15 measured over the whole scene.) Both actions, the matrix together with the white balance, 
J: offer the desired white reproduction. The Ro', Go', Bo' signals now guarantee an EBU color 

ill reproduction. 



For a proper near white luminance signal Yn the opposite has to be done, 
20 Therefore, imagine a scene with colors according to the EBU color space and a color 

temperature equal to D65 white. With the inverse matrix of the one shown below, the color 
space of the sensor is achieved: 
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where Rii, Gii, Bii represent the colors of an EBU scene and Ri,Gi, Bi the colors of the 
25 sensor. 

For the luminance signal Yn only the white reproduction of the inverse matrix 
is of interest, being represented by the sum of the matrix coefficients of each color. 
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Z/ta = 61 1 + 612 + 613 
mu = 621 + 622 + 623 
EM = 631 + 632 + 633 



In addition, the color temperature of the scene need not be D65 white. 
Inclusive an arbitrary color temperature the sum of the matrix coefficients becomes: 
XRiwb = Rpresetgain ■ 61 1 + Gpresetgain • 612 + Bpresetgain • 613 
ZGiwb = Rpresetgain • 621 + Gpresetgain • 622 + Bpresetgain ■ 623 
SB w6 = Rpresetgain • 63 1 + Gpresetgain ■ 632 + Bpresetgain • 633 



where Xpresetgain (X=R, G or B) represents the gain factors for transferring D65 white into 
that arbitrary color temperature. (For D65 white all Xpresetgain parameters are one.) 
To the SmartGcntrlR/B parameters used in Yn[i,j] and Yn[i+l,j+l] (see formula below) 



applies that: 

ZGiwb 



SmartGcntrlR - 
SmartGcntrlB = 



£/?w6 
EG/V6 
ZBiwb 



The parameter SGiwb is used as nominator because the green amplitude is 
regarded as a reference. This applies to the white balance as well. 

Now the above formulas can be written in such a way that the measured white 
balance parameters awbR/B may be applied. Knowing that 



Gpresetgain _ ^ _ Gtotal 

Rpresetgain Rtotal 

Gpresetgain , _ Gtotal 

2 = aw&S = 

Bpresetgain Btotal 

therefore 

vo- z. ^ ■ ( bU 701 613 ^ 
XRiwb = Gpresetgain • + 621 + 

^aw67? awbB J 

£Giw6 = Gpresetgain • f + 622 + &23 ] 

\^mv6/? aw6B J 



ILBiwb = Gpresetgain 



awbR awbB J 
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Since the ZXiwb- values are divided in the above, the parameter Gpresetgain is 
not important because Gpresetgain/Gpresetgain=l. Therefore, the next formula is sufficient 
for calculating the desired EXiwb- values: 



Utiwb = 



( Ml W3 
+ 621 + - 



awbR awbB 



2 f &21 700 623 ^ 
ZGiwb = I + 622 + 



ILBiwb = 



Now a luminance signal Yn has become available with equal RGB signal 
amplitudes for white scene colors due to the fact that the sensor matrix and the color 
temperature of the scene have been taken into account. 



H| 10 Fig. 9 shows the block diagram of the RGB reconstruction. Yn is the 

multiplexed RGB-signal of the sensor where R has been multiplied with SmartcntrlR, and B 
with SmartcntrlB, in a preprocessing block 5. This Yn-signal is used for the embodied 
Q smartgreen reconstruction. Via a zero switchbox ZSB, Yn is splitted into three colors, red = 

R*SmartcntrlR, green = G, and Blue = B*SmartcntrlB, making the conventional Laplacian 
15 RGB reconstruction possible with smart green in a 3x3 RGB reconstruction with/without 
smartgreen processing block 7. The so-called RBc signal in the median filter already fits 
R*SmartGcntrlR and B*SmartGcntrlB. 

By dividing the reconstructed red and blue signals in dividers Dr and Db by 
SmartGcntrlR and SmartGcntrlB, respectively, the original red (Ro) and blue (Bo) sensor 
20 amplitudes are restored. This means that the usually applied matrix, white balance and 
gamma functions can be maintained. In digital circuit design multipliers are preferred to 
dividers. Therefore, in order to avoid the divider circuits, the best way is to let the computer 
of the camera calculate 1/SmartcntrlR and 1/SmartcntrlB. Next, via two separate wires, those 
values can be offered to two multipliers. The Ro-amplitude now becomes equal to the R- 
25 amplitude of the input signal (SmartcntrlR*R*(l/SmartcntrlR=R). The very same applies to 
the Bo-amplitude. 

It should be noted that the parameters SmartcntrlR/B have been determined in 
a measurement cycle before the photograph is taken or in a continuous way in case of video 
mode. 
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The image sensor used for generating the array of pixels may form part of a 
digital camera. The pixels of the array may be arranged in perpendicular rows and columns; 
however, alternative arrangements may also be possible. 

In order to obtain full color images with high color resolution, the sensor may 
be covered with a mosaic-like color pattern, where each pixel of the sensor is positioned 
behind a given color filter - e.g. a green filter. The RGB Bayer mosaic color filter would be a 
preferred choice in the present invention. However the method could also be used with a 
YeGCy Bayer sensor. Then the SmartGcntrlR/B parameters are calculated as where Ye-G = 
Red and Cy-G = Blue: 

YtGiwb 



SmartGcntrlR = 
SmartGcntrlB — 



ZRiwb + XGiwb 

ZGiwb 
YSiwb + ZGiwb 



Although the present invention has been described in connection with the 
preferred embodiment, it is not intended to be limited to the specific form set forth herein. On 
the contrary, it is intended to cover such alternatives, modifications, and equivalents, as can 
be reasonably included within the scope of the invention as defined by the appended claims. 
In the claims, any reference signs placed between parentheses shall not be construed as 
limiting the claim. The word "comprising" does not exclude the presence of elements or steps 
other than those listed in a claim. The word "a" or "an" preceding an element does not 
exclude the presence of a plurality of such elements. The invention can be implemented by 
means of hardware comprising several distinct elements, and by means of a suitably 
programmed computer. In the device claim enumerating several means, several of these 
means can be embodied by one and the same item of hardware. The mere fact that certain 
measures are recited in mutually different dependent claims does not indicate that a 
combination of these measures cannot be used to advantage. 



